feat(inversion): sparse log det of the split-regularization H on the numpy path (#267 lever 2) - #554

Summary

Lever 2 of PyAutoLabs/autolens_profiling#267 (phase 3 of the fixed-lens-light-numba-cpu epic), stacked on lever 1 (#553): AbstractInversion.log_det_regularization_matrix_term no longer dense-factorises the split-regularization H on the numpy path.

After lever 1, inversion.log_det_regularization_matrix_term was ~37 ms of a ~300 ms production numba CPU likelihood call (HST Delaunay N=1500, AdaptSplit, InversionImagingSparseNumba, 1 thread): a dense O(P³) Cholesky of regularization_matrix_reduced, a (1500, 1500) SPD matrix with 8.45 non-zeros per row. The docstring claimed the term "uses scipy sparse linear algebra"; it did not.

What changed

• New inversion_util.log_det_sparse_spd_from(matrix) -> Optional[float]: one pass over the dense matrix builds the CSC triple directly (no csc_matrix(dense), which costs 13 ms), SuperLU in symmetric mode (permc_spec="MMD_AT_PLUS_A", diag_pivot_thresh=0.0), log det = sum(log(diag(U))). Returns None when the dense Cholesky is the faster route.
• Two measured regime switches, documented with their tables in the constants' comments:
  • SPARSE_LOG_DET_MAX_NNZ_PER_ROW = 32: geometric-stencil H (neighbour and split schemes, ~5–30 nnz/row, planar-graph structure) goes sparse; kernel H (MaternKernel and siblings, fully dense) stays dense. Real Matern P=1500: dense 34.9 ms vs forced-sparse 335 ms, so the gate matters.
  • SPARSE_LOG_DET_MIN_PIXELS = 256: SuperLU's fixed setup loses below that (sparse/dense 0.38x at P=128, 1.8x at 256, 10.8x at 1500).
• _log_det_symmetric_from(matrix, sparse=False) offers the matrix to the helper first when sparse=True; log_det_regularization_matrix_term passes sparse=self._xp is np. The "slogdet" opt-in path, the JAX path, and log_det_curvature_reg_matrix_term (F + λH, dense, lever 3) are byte-identical to lever 1.
• Failure semantics unchanged: a non-positive pivot or an exactly singular factor raises np.linalg.LinAlgError, exactly as np.linalg.cholesky, so the test-mode guard applies as before. No fallback, no warning-and-continue.
• Stale docstring at log_det_regularization_matrix_term replaced with what runs on each backend.

API Changes

None in signature or exceptions. Value change on the numpy path only, at round-off level: the sparse LU and the dense Cholesky are two fp64 factorisations of a matrix with condition number 6.4e12 (the 2e-8 diagonal jitter against the stencil's smooth null modes), so they agree to 7.4e-10 relative on the log-det term (−5.7e-6 nats) and 9.8e-11 relative on the fit's log evidence (−2.9e-6 nats) at HST Delaunay N=1500 — inside the phase's 1e-9 pin. Below P=256 the value is bit-identical to before (dense path). The existing numpy-vs-JAX parity tests (P=9/16, pinned at 1e-10) are unaffected for that reason; for the record, at that size numpy-splu vs jax-dense differed by 4.3e-10 relative, i.e. a 1e-10 cross-backend pin is only met when both sides run the same algorithm.

Measurements

Local (1 thread, real AdaptSplit H, P=1500): dense Cholesky 36.9 ms → sparse 6.1 ms (of which 2.9 ms is the non-zero scan). RCM + banded Cholesky was tried and rejected (43 ms; the Hilbert ordering leaves a wide band).

RAL A/B (control = lever 1 bae9296e, feature = lever 2, gpu partition CPUs-only, 1 thread, phase-2 Leg D cell, route b memo ON):

1.13x on the whole call (1.135x on medians); the site falls from 12.4 % to 2.4 % of the call and nothing else moves (the sibling log_det_curvature_reg_matrix_term is flat at 40.2 → 39.6 ms). The CPU job ran alone on the node (load average at the timed rows 0.49 control, 0.78 feature, no second tenant), so this is a measurement, not a bound. The control reproduces lever 1's feature arm to +1.0 %. Cumulative levers 1+2 against the merge base: 413.3 → 267.4 ms, 1.55x.

Witness inside the feature arm on RAL: the sparse route fires exactly once per figure_of_merit; sparse vs dense-forced log det Δ −9.2e-6 nats (1.2e-9 relative, H eigenvalue ratio 6.5e12); figure_of_merit Δ −4.6e-6 nats (2.3e-10 relative, inside the phase's 1e-9 pin); both gates admit the matrix (P=1500, 8.49 nnz/row); same-matrix timing 6.27 vs 37.72 ms.

A100 row (job 343354, fixed_light.py Delaunay cell, fp64): identity on the JAX path — all twelve log_evidence_terms bit-identical between arms. The A100's own dense Cholesky of the same H costs 1.13 ms, 5.6x faster than the CPU sparse route, so this lever is CPU-only by measurement and is deliberately not carried to the GPU.

Testing

• New test_autoarray/inversion/inversion/test_log_det_sparse.py (17 tests): helper vs slogdet at rtol 1e-12 on seeded well-conditioned banded SPD matrices (cond < 1e6 asserted); vs dense Cholesky on real ConstantSplit (1e-10) and AdaptSplit (1e-7, with the dense-vs-slogdet spread asserted at the same tolerance so the bound is the matrix's fp64 ambiguity, not a choice) split-stencil H; real kernel H takes the dense route; small H takes the dense route; singular and non-PD matrices raise LinAlgError on both routes; "slogdet" path unchanged; the term on a real numpy-path inversion matches its dense value.
• pytest test_autoarray/inversion -q — 570 passed; pytest test_autoarray/inversion/inversion -q — 229 passed.

Tracked by PyAutoLabs/autolens_profiling#267 (lever 2 of 3). Merge order: #553 first.

Heart gate

pyauto-heart readiness was RED (score 20) at ship time with two release-chain reasons unrelated to this branch — "install verification FAILED (testpypi; checks F)" and "release validation FAILED (stage integrate)" — acknowledged by the human on 2026-09-16 for PR-open only (levers 2 and 3, same reasons as lever 1). Merge stays a human act via /prm.

🤖 Generated with Claude Code